[SPARK-22883][ML][TEST] Streaming tests for spark.ml.feature, from A to H by jkbradley · Pull Request #20111 · apache/spark

jkbradley · 2017-12-29T03:42:21Z

What changes were proposed in this pull request?

Adds structured streaming tests using testTransformer for these suites:

BinarizerSuite
BucketedRandomProjectionLSHSuite
BucketizerSuite
ChiSqSelectorSuite
CountVectorizerSuite
DCTSuite.scala
ElementwiseProductSuite
FeatureHasherSuite
HashingTFSuite

How was this patch tested?

It tests itself because it is a bunch of tests!

jkbradley · 2017-12-29T03:42:34Z

mllib/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala

No existing test to use

jkbradley · 2017-12-29T03:43:13Z

mllib/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala

Moved from object to class b/c this needed testTransformer from the MLTest mix-in

jkbradley · 2017-12-29T03:44:04Z

mllib/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala

No existing unit test to use

jkbradley · 2017-12-29T03:44:18Z

mllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala

scala style

SparkQA · 2017-12-29T04:50:38Z

Test build #85494 has finished for PR 20111 at commit 12b3dcf.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinarizerSuite extends MLTest with DefaultReadWriteTest
class BucketedRandomProjectionLSHSuite extends MLTest with DefaultReadWriteTest
class BucketizerSuite extends MLTest with DefaultReadWriteTest
class ChiSqSelectorSuite extends MLTest with DefaultReadWriteTest
class CountVectorizerSuite extends MLTest with DefaultReadWriteTest
class DCTSuite extends MLTest with DefaultReadWriteTest
class ElementwiseProductSuite extends MLTest with DefaultReadWriteTest
class FeatureHasherSuite extends MLTest with DefaultReadWriteTest
class HashingTFSuite extends MLTest with DefaultReadWriteTest

jkbradley · 2017-12-29T06:00:02Z

mllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala

Rearranged this test so it checks each row independently.

jkbradley · 2017-12-29T06:00:34Z

mllib/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala

ditto: rearranged to do validity check per-row

WeichenXu123 · 2017-12-29T10:45:02Z

mllib/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala

So here do not need to select "keys" column ?

I don't think we have to. The main thing here is to make sure that the transform really does happen. Other tests check validity of the values.

OK. so I prefer to use simpler code:

testTransformer[Tuple1[Vector]](dataset.toDF(), brpModel, "values") { case Row(values: Seq[_]) => ...

WeichenXu123 · 2017-12-30T04:20:12Z

LGTM except a tiny issue. :)

SparkQA · 2018-01-08T09:19:12Z

Test build #85789 has finished for PR 20111 at commit 12b3dcf.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BinarizerSuite extends MLTest with DefaultReadWriteTest
class BucketedRandomProjectionLSHSuite extends MLTest with DefaultReadWriteTest
class BucketizerSuite extends MLTest with DefaultReadWriteTest
class ChiSqSelectorSuite extends MLTest with DefaultReadWriteTest
class CountVectorizerSuite extends MLTest with DefaultReadWriteTest
class DCTSuite extends MLTest with DefaultReadWriteTest
class ElementwiseProductSuite extends MLTest with DefaultReadWriteTest
class FeatureHasherSuite extends MLTest with DefaultReadWriteTest
class HashingTFSuite extends MLTest with DefaultReadWriteTest

jkbradley · 2018-03-01T22:04:30Z

Updated! Thanks @WeichenXu123 -- I'll merge this once tests pass.

SparkQA · 2018-03-01T23:11:01Z

Test build #87857 has finished for PR 20111 at commit 448668d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2018-03-02T02:40:27Z

Merging with master and branch-2.3

…to H ## What changes were proposed in this pull request? Adds structured streaming tests using testTransformer for these suites: * BinarizerSuite * BucketedRandomProjectionLSHSuite * BucketizerSuite * ChiSqSelectorSuite * CountVectorizerSuite * DCTSuite.scala * ElementwiseProductSuite * FeatureHasherSuite * HashingTFSuite ## How was this patch tested? It tests itself because it is a bunch of tests! Author: Joseph K. Bradley <joseph@databricks.com> Closes #20111 from jkbradley/SPARK-22883-streaming-featureAM. (cherry picked from commit 119f6a0) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>

…to H ## What changes were proposed in this pull request? Adds structured streaming tests using testTransformer for these suites: * BinarizerSuite * BucketedRandomProjectionLSHSuite * BucketizerSuite * ChiSqSelectorSuite * CountVectorizerSuite * DCTSuite.scala * ElementwiseProductSuite * FeatureHasherSuite * HashingTFSuite ## How was this patch tested? It tests itself because it is a bunch of tests! Author: Joseph K. Bradley <joseph@databricks.com> Closes apache#20111 from jkbradley/SPARK-22883-streaming-featureAM. (cherry picked from commit 119f6a0) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>

jkbradley commented Dec 29, 2017

View reviewed changes

mllib/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala Outdated

Copy link

Member Author

jkbradley Dec 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No existing test to use

jkbradley commented Dec 29, 2017

View reviewed changes

mllib/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala Outdated

Copy link

Member Author

jkbradley Dec 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No existing unit test to use

jkbradley commented Dec 29, 2017

View reviewed changes

mllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala Outdated

Copy link

Member Author

jkbradley Dec 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scala style

jkbradley commented Dec 29, 2017

View reviewed changes

mllib/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala Outdated

Copy link

Member Author

jkbradley Dec 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto: rearranged to do validity check per-row

WeichenXu123 reviewed Dec 29, 2017

View reviewed changes

jkbradley added 2 commits March 1, 2018 14:03

added streaming tests for first quarter of spark.ml.feature

551befd

simplified code per review

448668d

jkbradley force-pushed the SPARK-22883-streaming-featureAM branch from 12b3dcf to 448668d Compare March 1, 2018 22:03

asfgit closed this in 119f6a0 Mar 2, 2018

jkbradley deleted the SPARK-22883-streaming-featureAM branch March 2, 2018 06:28

Conversation

jkbradley commented Dec 29, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 29, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WeichenXu123 commented Dec 30, 2017

Uh oh!

SparkQA commented Jan 8, 2018

Uh oh!

jkbradley commented Mar 1, 2018

Uh oh!

SparkQA commented Mar 1, 2018

Uh oh!

jkbradley commented Mar 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

jkbradley commented Mar 2, 2018 •

edited

Loading